{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Prediction with a custom scikit-learn pipeline",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "KXNKiFZTUySu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Copyright 2019 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "137G11XIWgCy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Prediction with a custom scikit-learn pipeline\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://cloud.google.com/ml-engine/docs/scikit/custom-pipeline\">\n",
        "      <img src=\"https://cloud.google.com/_static/images/cloud/icons/favicons/onecloud/super_cloud.png\"\n",
        "           alt=\"Google Cloud logo\" width=\"32px\"> Read on cloud.google.com\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/cloudml-samples/blob/master/notebooks/scikit-learn/custom-pipeline.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/notebooks/scikit-learn/custom-pipeline.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "metadata": {
        "id": "KXEex-vKeSkC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Beta**\n",
        "\n",
        "This is a beta release of custom code for scikit-learn pipelines. This feature might be changed in backward-incompatible ways and is not subject to any SLA or deprecation policy."
      ]
    },
    {
      "metadata": {
        "id": "tW9YCwlGY63D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial shows how to use AI Platform to deploy a scikit-learn pipeline that uses custom transformers.\n",
        "\n",
        "[scikit-learn pipelines](https://scikit-learn.org/stable/modules/compose.html) allow you to compose multiple estimators. For example, you can use transformers to preprocess data and pass the transformed data to a classifier. scikit-learn provides [many\n",
        "transformers](https://scikit-learn.org/stable/data_transforms.html) in the `sklearn` package.\n",
        "\n",
        "You can also use scikit-learn's [`FunctionTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html) or [`TransformerMixin`](https://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html) class to create your own custom transformer. If you want to deploy a pipeline that uses custom transformers to AI Platform Prediction, you must [provide that code to AI Platform as a source distribution package](https://cloud.google.com/ml-engine/docs/scikit/exporting-for-prediction#custom-pipeline-code).\n",
        "\n",
        "This tutorial presents a sample problem involving Census data to walk you through the following steps:\n",
        "\n",
        "* Training a scikit-learn pipeline with custom transformers on AI Platform Training\n",
        "* Deploying the trained pipeline and your custom code to AI\n",
        "Platform Prediction\n",
        "* Serving prediction requests from that deployment"
      ]
    },
    {
      "metadata": {
        "id": "Pd0wS_kKY_Du",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Dataset\n",
        "\n",
        "This tutorial uses the [United States Census Income\n",
        "Dataset](https://archive.ics.uci.edu/ml/datasets/census+income) provided by the\n",
        "[UC Irvine Machine Learning\n",
        "Repository](https://archive.ics.uci.edu/ml/index.php). This dataset contains\n",
        "information about people from a 1994 Census database, including age, education,\n",
        "marital status, occupation, and whether they make more than $50,000 a year.\n",
        "\n",
        "The data used in this tutorial is available in a public Cloud Storage bucket:\n",
        "[`gs://cloud-samples-data/ml-engine/sklearn/census_data/`](https://console.cloud.google.com/storage/browser/cloud-samples-data/ml-engine/sklearn/census_data/)"
      ]
    },
    {
      "metadata": {
        "id": "2OFY1EMTZAfI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Objective\n",
        "\n",
        "The goal is to train a scikit-learn pipeline that predicts\n",
        "whether a person makes more than $50,000 a year (target label) based on other\n",
        "Census information about the person (features).\n",
        "\n",
        "This tutorial focuses more on using this model with AI Platform than on\n",
        "the design of the model itself. However, it's always important to think about\n",
        "potential problems and unintended consequences when building machine learning\n",
        "systems. See the [Machine Learning Crash Course exercise about\n",
        "fairness](https://developers.google.com/machine-learning/crash-course/fairness/programming-exercise)\n",
        "to learn about sources of bias in the Census dataset, as well as machine\n",
        "learning fairness more generally."
      ]
    },
    {
      "metadata": {
        "id": "GIU-iw6WZRJO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
        "\n",
        "* AI Platform\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [AI Platform\n",
        "pricing](https://cloud.google.com/ml-engine/docs/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "metadata": {
        "id": "91gDvdNHZX5S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Before you begin\n",
        "\n",
        "You must do several things before you can train and deploy a model in AI Platform:\n",
        "\n",
        "* Set up your local development environment.\n",
        "* Set up a GCP project with billing and the necessary\n",
        "  APIs enabled.\n",
        "* Authenticate your GCP account in this notebook.\n",
        "* Create a Cloud Storage bucket to store your training package and your\n",
        "  trained model.\n"
      ]
    },
    {
      "metadata": {
        "id": "VC2c7_HJZZT_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "**If you are using Colab or AI Platform Notebooks**, your environment already\n",
        "meets all the requirements to run this notebook. You can skip this step."
      ]
    },
    {
      "metadata": {
        "id": "VwDsbQE0a0Mp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
        "You need the following:\n",
        "\n",
        "* The Google Cloud SDK\n",
        "* Python 3\n",
        "* virtualenv\n",
        "* Jupyter notebook running in a virtual environment with Python 3\n",
        "\n",
        "The Google Cloud guide to [Setting up a Python development\n",
        "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
        "installation guide](https://jupyter.org/install) provide detailed instructions\n",
        "for meeting these requirements. The following steps provide a condensed set of\n",
        "instructions:\n",
        "\n",
        "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
        "\n",
        "2. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
        "\n",
        "3. [Install\n",
        "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
        "   and create a virtual environment that uses Python 3. (You can skip this step    if you want to install Jupyter globally.)\n",
        "\n",
        "4. Activate that environment and run `pip install jupyter` in a shell to install\n",
        "   Jupyter.\n",
        "\n",
        "5. Run `jupyter notebook` in a shell to launch Jupyter.\n",
        "\n",
        "6. Open this notebook in the Jupyter Notebook Dashboard."
      ]
    },
    {
      "metadata": {
        "id": "ldy2Z_z-a-0z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Set up your GCP project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager)\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the AI Platform (\"Cloud Machine Learning Engine\") and Compute Engine\n",
        "APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
        "\n",
        "4. Enter your project ID below and run the following cells to make sure the Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "metadata": {
        "id": "wsuRZACjbiES",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "PROJECT_ID = \"<your-project-id>\" #@param {type:\"string\"}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CuucDvG6NyT4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! gcloud config set project $PROJECT_ID"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "x9hqlEa-bpIX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Authenticate your GCP account\n",
        "\n",
        "**If you are using AI Platform Notebooks**, your environment is already\n",
        "authenticated. Skip this step."
      ]
    },
    {
      "metadata": {
        "id": "QvekLkQibra5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**If you are using Colab**, run the cell below and follow the instructions\n",
        "when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "1. In the GCP Console, go to the [**Create service account key**\n",
        "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
        "\n",
        "2. From the **Service account** drop-down list, select **New service account**.\n",
        "\n",
        "3. In the **Service account name** field, enter a name.\n",
        "\n",
        "4. From the **Role** drop-down list, select\n",
        "   **Machine Learning Engine > AI Platform Admin** and\n",
        "   **Storage > Storage Object Admin**.\n",
        "\n",
        "5. Click *Create*. A JSON file that contains your key downloads to your\n",
        "local environment.\n",
        "\n",
        "6. Enter the path to your service account key as the\n",
        "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
      ]
    },
    {
      "metadata": {
        "id": "qkL641qHb1SZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import sys\n",
        "\n",
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "if 'google.colab' in sys.modules:\n",
        "  from google.colab import auth as google_auth\n",
        "  google_auth.authenticate_user()\n",
        "\n",
        "# If you are running this notebook locally, replace the string below with the\n",
        "# path to your service account key and run this cell to authenticate your GCP\n",
        "# account.\n",
        "else:\n",
        "  %env GOOGLE_APPLICATION_CREDENTIALS '<path-to-your-service-account-key.json>'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "O4E2pjYacgtz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "This tutorial uses Cloud Storage in several ways:\n",
        "\n",
        "* When you submit a training job using the Cloud SDK, you upload a Python\n",
        "  package containing your training code to a Cloud Storage bucket. AI Platform\n",
        "  runs the code from this package.\n",
        "\n",
        "* In this tutorial, AI Platform also saves the trained model that results from\n",
        "  your job in the same bucket.\n",
        "\n",
        "* To deploy your scikit-learn pipeline that uses custom code to serve\n",
        "  predictions, you must upload the custom transformers that your pipeline uses\n",
        "  to Cloud Storage.\n",
        "\n",
        "When you create the AI Platform version resource that serves predictions, you\n",
        "provide the trained scikit-learn pipeline and your custom code as Cloud Storage\n",
        "URIs.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
        "Cloud Storage buckets. \n",
        "\n",
        "You may also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook. Make sure to [choose a region where Cloud\n",
        "AI Platform services are\n",
        "available](https://cloud.google.com/ml-engine/docs/scikit/regions). You may\n",
        "not use a Multi-Regional Storage bucket for training with AI Platform."
      ]
    },
    {
      "metadata": {
        "id": "31wtdj9od-3R",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "BUCKET_NAME = \"<your-bucket-name>\" #@param {type:\"string\"}\n",
        "REGION = \"us-central1\" #@param {type:\"string\"}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3EG9EzUOeBPI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "metadata": {
        "id": "HWLSNArseDWH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! gsutil mb -l $REGION gs://$BUCKET_NAME"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BjW8HyEDeGlM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:\n"
      ]
    },
    {
      "metadata": {
        "id": "XpORnbjPeMpK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! gsutil ls -al gs://$BUCKET_NAME"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NKQvwF34NaVO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Creating a training application and custom pipeline code"
      ]
    },
    {
      "metadata": {
        "id": "Evt9r66SeDYS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Create an application to train a scikit-learn pipeline with the\n",
        "Census data. In this tutorial, the training package also contains the custom\n",
        "code that the trained pipeline uses during prediction. This is a useful\n",
        "pattern, because pipelines are generally designed to use the same transformers during training and prediction.\n",
        "\n",
        "Use the following steps to create a directory with three files inside that matches the following structure:\n",
        "\n",
        "```\n",
        "census_package/\n",
        "    __init__.py\n",
        "    my_pipeline.py\n",
        "    train.py\n",
        "```\n",
        "\n",
        "First, create the empty `census_package/` directory:"
      ]
    },
    {
      "metadata": {
        "id": "w_eBjogAuz6_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! mkdir census_package"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8U6uNGaPfdnI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Within `census_package/` create a blank file named `__init__.py` :"
      ]
    },
    {
      "metadata": {
        "id": "8FnCC9u-iq8q",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! touch ./census_package/__init__.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TuZsxj34r8B-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This makes it possible to import `census_package/` as a package in Python."
      ]
    },
    {
      "metadata": {
        "id": "laIEwRPnsDdT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Create custom transformers"
      ]
    },
    {
      "metadata": {
        "id": "CUKKvr6Lh-78",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "scikit-learn provides [many transformers](https://scikit-learn.org/stable/data_transforms.html) that you can use as part of a pipeline, but it also lets\n",
        "you define your own custom transformers. These transformers can even learn a\n",
        "saved state during training that gets used later during prediction.\n",
        "\n",
        "Extend [`sklearn.base.TransformerMixin`](https://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html) to create several custom transformers in a file named `census_package/my_pipeline.py`.\n",
        "\n",
        "Run the following cell to define three transformers:\n",
        "\n",
        "* `PositionalSelector`: Given a list of indices C and a matrix M, this returns a matrix with a subset of M's columns, indicated by C.\n",
        "\n",
        "* `StripString`: Given a matrix of strings, this strips whitespaces from each string.\n",
        "\n",
        "* `SimpleOneHotEncoder`: A simple [one-hot encoder](https://developers.google.com/machine-learning/glossary/#o) which can be applied to a matrix of strings."
      ]
    },
    {
      "metadata": {
        "id": "L4lM03iH95hv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%writefile ./census_package/my_pipeline.py\n",
        "import numpy as np\n",
        "from sklearn.base import BaseEstimator, TransformerMixin\n",
        "\n",
        "\n",
        "class PositionalSelector(BaseEstimator, TransformerMixin):\n",
        "    def __init__(self, positions):\n",
        "        self.positions = positions\n",
        "\n",
        "    def fit(self, X, y=None):\n",
        "        return self\n",
        "\n",
        "    def transform(self, X):\n",
        "        return np.array(X)[:, self.positions]\n",
        "\n",
        "\n",
        "class StripString(BaseEstimator, TransformerMixin):\n",
        "    def fit(self, X, y=None):\n",
        "        return self\n",
        "\n",
        "    def transform(self, X):\n",
        "        strip = np.vectorize(str.strip)\n",
        "        return strip(np.array(X))\n",
        "\n",
        "\n",
        "class SimpleOneHotEncoder(BaseEstimator, TransformerMixin):\n",
        "    def fit(self, X, y=None):\n",
        "        self.values = []\n",
        "        for c in range(X.shape[1]):\n",
        "            Y = X[:, c]\n",
        "            values = {v: i for i, v in enumerate(np.unique(Y))}\n",
        "            self.values.append(values)\n",
        "        return self\n",
        "\n",
        "    def transform(self, X):\n",
        "        X = np.array(X)\n",
        "        matrices = []\n",
        "        for c in range(X.shape[1]):\n",
        "            Y = X[:, c]\n",
        "            matrix = np.zeros(shape=(len(Y), len(self.values[c])), dtype=np.int8)\n",
        "            for i, x in enumerate(Y):\n",
        "                if x in self.values[c]:\n",
        "                    matrix[i][self.values[c][x]] = 1\n",
        "            matrices.append(matrix)\n",
        "        res = np.concatenate(matrices, axis=1)\n",
        "        return res"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "X2f93oDDvsFb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note: You can also create custom transformers by using [`sklearn.preprocessing.FunctionTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html), but this only works for stateless transformations.\n"
      ]
    },
    {
      "metadata": {
        "id": "OxPi36JRiUPy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Define pipeline and create training module\n",
        "\n",
        "Next, create a training module to train your scikit-learn pipeline on Census\n",
        "data. Part of this code involves defining the pipeline.\n",
        "\n",
        "This training module does several things:\n",
        "\n",
        "* It downloads training data and loads it into a pandas `DataFrame` that can be used by scikit-learn.\n",
        "* It defines the scikit-learn pipeline to train. This examples only uses three numerical features (`'age'`, `'education-num'`, and `'hours-per-week'`) and three categorical features (`'workclass'`, `'marital-status'`, and `'relationship'`) from the input data. It transforms the numerical features using scikit-learn's built-in [`StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) and transforms the categorical ones with the custom one-hot encoder you defined in `my_pipeline.py`. Then it combines the preprocessed data as input for a classifier.\n",
        "* Finally, it exports the model using `joblib` and saves it to your Cloud Storage bucket.\n",
        "\n",
        "Run the following cell to create `census_package/train.py`:"
      ]
    },
    {
      "metadata": {
        "id": "0DtxCmp1uz7C",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%writefile ./census_package/train.py\n",
        "import warnings\n",
        "import argparse\n",
        "from google.cloud import storage\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.externals import joblib\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "from sklearn.pipeline import Pipeline, FeatureUnion, make_pipeline\n",
        "import census_package.my_pipeline as mp\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "\n",
        "def download_data(bucket_name, gcs_path, local_path):\n",
        "    bucket = storage.Client().bucket(bucket_name)\n",
        "    blob = bucket.blob(gcs_path)\n",
        "    blob.download_to_filename(local_path)\n",
        "\n",
        "\n",
        "def upload_data(bucket_name, gcs_path, local_path):\n",
        "    bucket = storage.Client().bucket(bucket_name)\n",
        "    blob = bucket.blob(gcs_path)\n",
        "    blob.upload_from_filename(local_path)\n",
        "\n",
        "\n",
        "def get_features_target(local_path):\n",
        "    strip = np.vectorize(str.strip)\n",
        "    raw_df = pd.read_csv(local_path, header=None)\n",
        "    target_index = len(raw_df.columns) - 1  # Last columns, 'income-level', is the target\n",
        "\n",
        "    features_df = raw_df.drop(target_index, axis=1)\n",
        "    features = features_df.as_matrix()\n",
        "    target = strip(raw_df[target_index].values)\n",
        "    return features, target\n",
        "\n",
        "\n",
        "def create_pipeline():\n",
        "    # We want to use 3 categorical and 3 numerical features in this sample.\n",
        "    # Categorical features: age, education-num, and hours-per-week\n",
        "    # Numerical features: workclass, marital-status, and relationship\n",
        "    numerical_indices = [0, 4, 12]  # age, education-num, and hours-per-week\n",
        "    categorical_indices = [1, 5, 7]  # workclass, marital-status, and relationship\n",
        "\n",
        "    p1 = make_pipeline(mp.PositionalSelector(categorical_indices), mp.StripString(), mp.SimpleOneHotEncoder())\n",
        "    p2 = make_pipeline(mp.PositionalSelector(numerical_indices), StandardScaler())\n",
        "\n",
        "    feats = FeatureUnion([\n",
        "        ('numericals', p1),\n",
        "        ('categoricals', p2),\n",
        "    ])\n",
        "\n",
        "    pipeline = Pipeline([\n",
        "        ('pre', feats),\n",
        "        ('estimator', GradientBoostingClassifier(max_depth=4, n_estimators=100))\n",
        "    ])\n",
        "    return pipeline\n",
        "\n",
        "\n",
        "def get_bucket_path(gcs_uri):\n",
        "    if not gcs_uri.startswith('gs://'):\n",
        "        raise Exception('{} does not start with gs://'.format(gcs_uri))\n",
        "    no_gs_uri = gcs_uri[len('gs://'):]\n",
        "    first_slash_index = no_gs_uri.find('/')\n",
        "    bucket_name = no_gs_uri[:first_slash_index]\n",
        "    gcs_path = no_gs_uri[first_slash_index + 1:]\n",
        "    return bucket_name, gcs_path\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    parser = argparse.ArgumentParser()\n",
        "    parser.add_argument('--gcs_data_path', action=\"store\", required=True)\n",
        "    parser.add_argument('--gcs_model_path', action=\"store\", required=True)\n",
        "\n",
        "    arguments, others = parser.parse_known_args()\n",
        "\n",
        "    local_path = '/tmp/adul.data'\n",
        "    data_bucket, data_path = get_bucket_path(arguments.gcs_data_path)\n",
        "    print('Downloading the data...')\n",
        "    download_data(data_bucket, data_path, local_path)\n",
        "    features, target = get_features_target(local_path)\n",
        "    pipeline = create_pipeline()\n",
        "    \n",
        "    print('Training the model...')\n",
        "    pipeline.fit(features, target)\n",
        "    \n",
        "    joblib.dump(pipeline, './model.joblib')\n",
        "    \n",
        "    model_bucket, model_path = get_bucket_path(arguments.gcs_model_path)\n",
        "    upload_data(model_bucket, model_path, './model.joblib')\n",
        "    print('Model was successfully uploaded.')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t2R_jZ_BpHVO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Training the pipeline on AI Platform\n",
        "\n",
        "Use `gcloud` to submit a training job to AI Platform. The following command packages your training application, uploads it to Cloud Storage, and tells\n",
        "AI Platform to run your training module.\n",
        "\n",
        "The `--` argument is a separator: the AI Platform service doesn't use arguments\n",
        "that follow the separator, but your training module can still access them."
      ]
    },
    {
      "metadata": {
        "id": "zvUBWJDfkOkE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! gcloud ai-platform jobs submit training census_training_$(date +\"%Y%m%d_%H%M%S\") \\\n",
        "  --job-dir gs://$BUCKET_NAME/custom_pipeline_tutorial/job \\\n",
        "  --package-path ./census_package \\\n",
        "  --module-name census_package.train \\\n",
        "  --region $REGION \\\n",
        "  --runtime-version 1.13 \\\n",
        "  --python-version 3.5 \\\n",
        "  --scale-tier BASIC \\\n",
        "  --stream-logs \\\n",
        "  -- \\\n",
        "  --gcs_data_path gs://cloud-samples-data/ml-engine/census/data/adult.data.csv \\\n",
        "  --gcs_model_path gs://$BUCKET_NAME/custom_pipeline_tutorial/model/model.joblib"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "a2NROYJF2mlQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Deploying the pipeline and serving predictions\n",
        "\n",
        "To serve predictions from AI Platform, you must deploy a *model* resource and a\n",
        "*version* resource. The *model* helps you organize multiple deployments\n",
        "if you modify and train your pipeline multiple times. The *version* uses your\n",
        "trained model and custom code to serve predictions.\n",
        "\n",
        "To deploy these resources, you need to provide two artifacts:\n",
        "\n",
        "* A Cloud Storage directory containing your trained pipeline. The training job\n",
        "  from the previous step created this file when it exported `model.joblib` to\n",
        "  your bucket.\n",
        "* A `.tar.gz` source distribution package in Cloud Storage containing any custom\n",
        "  transformers your pipeline uses. Create this in the next step."
      ]
    },
    {
      "metadata": {
        "id": "6UHBAkHG6Ztj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Package your custom transformers\n",
        "\n",
        "If you deploy a version without providing the code from `my_pipeline.py`, AI Platform Prediction won't be able to import the custom transformers (for example, `mp.SimpleOneHotEncoder`) and it will be unable to serve predictions.\n",
        "\n",
        "Create the following `setup.py` to define a source distribution package for your code:"
      ]
    },
    {
      "metadata": {
        "id": "MIKzYwqQuz7E",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%writefile ./setup.py\n",
        "import setuptools\n",
        "setuptools.setup(name='census_package',\n",
        "      packages=['census_package'],\n",
        "      version=\"1.0\",\n",
        "      )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PM5DzQ-56oRJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Then run the following command to create `dist/census_package-1.0.tar.gz`:"
      ]
    },
    {
      "metadata": {
        "id": "Wju0YtTouz7H",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! python setup.py sdist --formats=gztar"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Nb89WfDi7AL_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note: This package contains `train.py` as well, but the version resource doesn't require it.\n",
        "\n",
        "Finally, upload this tarball to your Cloud Storage bucket:"
      ]
    },
    {
      "metadata": {
        "id": "UIZsOdUhi-Oh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! gsutil cp ./dist/census_package-1.0.tar.gz gs://$BUCKET_NAME/custom_pipeline_tutorial/code/census_package-1.0.tar.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Q9etrAYx7rk7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Create model and version resources"
      ]
    },
    {
      "metadata": {
        "id": "B5Affn2G7uGg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "First, define model and version names:"
      ]
    },
    {
      "metadata": {
        "id": "5XnONR6eqg9l",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "MODEL_NAME = 'CensusPredictor'\n",
        "VERSION_NAME = 'v1'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2wsXbnhmqosE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Then use the following command to create the model resource:"
      ]
    },
    {
      "metadata": {
        "id": "TjtBj2Ztqoyk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! gcloud ai-platform models create $MODEL_NAME \\\n",
        "  --regions $REGION"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7E-KITBBsEpx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally, create the version resource by providing Cloud Storage paths to your model directory (the one that contains `model.joblib`) and your custom code (`census_package-1.0.tar.gz`):"
      ]
    },
    {
      "metadata": {
        "id": "2wC-zFSiMZHc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# --quiet automatically installs the beta component if it isn't already installed \n",
        "! gcloud --quiet beta ai-platform versions create $VERSION_NAME --model $MODEL_NAME \\\n",
        "  --origin gs://$BUCKET_NAME/custom_pipeline_tutorial/model/ \\\n",
        "  --runtime-version 1.13 \\\n",
        "  --python-version 3.5 \\\n",
        "  --framework SCIKIT_LEARN \\\n",
        "  --package-uris gs://$BUCKET_NAME/custom_pipeline_tutorial/code/census_package-1.0.tar.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "soCVZR1H8mQK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Serving online predictions\n",
        "\n",
        "Try out your deployment by sending an online prediction request. First, install the Google APIs Client Library for Python:"
      ]
    },
    {
      "metadata": {
        "id": "2SGwfntsrEYs",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "! pip install --upgrade google-api-python-client"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SJyorPp3rEAw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Then send two instances of Census data to your deployed version:"
      ]
    },
    {
      "metadata": {
        "id": "cSt7ol37_FuB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import googleapiclient.discovery\n",
        "\n",
        "instances = [\n",
        "  [39, 'State-gov', 77516, ' Bachelors .  ', 13, 'Never-married', 'Adm-clerical', 'Not-in-family',\n",
        "   'White', 'Male', 2174, 0, 40, 'United-States', '<=50K'],\n",
        "  [50, 'Self-emp-not-inc', 83311, 'Bachelors', 13, 'Married-civ-spouse', 'Exec-managerial', 'Husband',\n",
        "   'White', 'Male', 0, 0, 13, 'United-States', '<=50K']\n",
        "]\n",
        "\n",
        "service = googleapiclient.discovery.build('ml', 'v1')\n",
        "name = 'projects/{}/models/{}/versions/{}'.format(PROJECT_ID, MODEL_NAME, VERSION_NAME)\n",
        "\n",
        "response = service.projects().predict(\n",
        "    name=name,\n",
        "    body={'instances': instances}\n",
        ").execute()\n",
        "\n",
        "if 'error' in response:\n",
        "    raise RuntimeError(response['error'])\n",
        "else:\n",
        "  print(response['predictions'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PzwzYgoV-Bjk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note: This code uses the credentials you set up during the authentication step to make the online prediction request.\n",
        "\n",
        "The version passes the input data through the trained pipeline and returns the classifier's results: either `<=50K` or `>50K` for each instance, depending on its prediction for the person's income bracket."
      ]
    },
    {
      "metadata": {
        "id": "kLX2V3TyswzH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all GCP resources used in this project, you can [delete the GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Alternatively, you can clean up individual resources by running the following commands:"
      ]
    },
    {
      "metadata": {
        "id": "ToGeA0Htssrl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Delete version resource\n",
        "! gcloud ai-platform versions delete $VERSION_NAME --quiet --model $MODEL_NAME\n",
        "\n",
        "# Delete model resource\n",
        "! gcloud ai-platform models delete $MODEL_NAME --quiet\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "! gsutil -m rm -r gs://$BUCKET_NAME/custom_pipeline_tutorial"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "i3inFBtMddy3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## What's next\n",
        "\n",
        "* Read more about [how to use custom scikit-learn pipelines](https://cloud.google.com/ml-engine/docs/scikit/exporting-for-prediction#custom-pipeline-code) with AI Platform Prediction.\n",
        "* Learn about creating a [custom prediction routine (beta)](https://cloud.google.com/ml-engine/docs/scikit/custom-prediction-routines) for even more control over how AI Platform serves predictions."
      ]
    }
  ]
}